Web Scraping

HTML, CSS & JAVASCRIPT

A web page typically is made up of the following:

• HTML (Hyper Text Markup Language) takes care of the content. You need to have a basic knowledge of HTML tags as the content is located with these tags.
• CSS (Cascading Style Sheets) takes care of the appearance of the contente. While you don’t need to look into the CSS of a web page, you should be able to identify the id or class that manage the appearance of content.
• JS (Javascript) takes care of the behavior of the web page.

HTML Element

HTML element consists of a start tag and end tag with content inserted in between. They can be nested and are case insensitive.

HTML Tags

Below is a list of basic and important HTML tags you should know before you get started with web scraping.

DOM

DOM (Document Object Model) defines the logical structure of a document and the way it is accessed and manipulated. In the above image, you can that the HTML is structured as a tree and you trace path to any node or tag. We will use a similar approach in our case studies.

HTML Attributes

• all HTML elements can have attributes
• they provide additional information about an element
• they are always specified in the start tag
• usually come in name/value pairs

The class attribute is used to define equal styles for elements with same class name. HTML elements with same class name will have the same format and style. The id attribute specifies a unique id for an HTML element. It can be used on any HTML element and is case sensitive. The style attribute sets the style of an HTML element.

Brand Name

The first detail we want to extract is the brand name of the phone. If you look at the HTML code, it is nested within a hyperlink, defined by <a>. The link is inside a section identified by the class crwTitle. We will use this information to ask rvest to extract text content which will give us the brand name.

The location is specified using html_nodes() and the text extracted using hmtl_text(). Since crwTitle is a class, we use . before it but not for a as it is a HTML tag. Both the class and the tag are specified with quotes and separated by space.

If you observe the output, it includes the following:

• brand name
• color
• RAM
• storage capacity

To extract the brand name, we will use str_split() from stringr and specify the pattern \\( i.e split the string @ the first opening bracket. Since ( is a special character, we use \\ for escaping. Next, we use map_chr() from the purrr package to extract the first element from the resulting list. Finally, we remove the white space using str_trim().

The whole point of the above exercise is to show that extracting the data using rvest is just one part of web scraping. We need to spend enough time tidying and reshaping the data to get it into a format useful for data analysis.

Color

In the previous step, we observed that the data extracted from top_mobiles included the color of the mobile as well. So the location of the color in the HTML is same; within the hyperlink of the crwTitle section. But now, we want to extract the color and not the brand name.

We will split the original string @ ( and extract the second part which includes:

• color
• RAM
• storage capacity

The color is separated from the rest by a comma. We will use the , to split the string and extract the color using map_chr() i.e. extract the first element from the resulting list.

Rating

Let us extract the ratings for the phones now. If you look at the HTML code, we can locate rating within the following:

• <span>
• <a>
• .crwProductDetail

It is wrapped within <span> identified by the class .a-icon-alt which is inside a hyperlink in the section identified by the class .crwProductDetail.

In the outptut, you can observe the text out of 5 stars for each rating. Let us get rid of this text by selecting the first 3 characters using str_sub(). We pick the first 3 characters using the start and end arguments and supply them the values 1 and 3. Finally, we convert the rating to a number using as.numeric().

Number of Reviews

Now that we know the rating for each of the top 10 best selling smart phones, let us find out how many people have reviewed them. The number of reviews is located within the following:

• hyperlink identified by the class .a-link-normal
• <span> tag identified by the class .a-size-small
• section identified by the class .crwProductDetail

We use the above information within html_nodes() to extract the data. Now let us clean it up a bit and convert it into a number instead of leaving it as a character. If you use as.numeric() directly, you will see NA in the result, the reason being the presence of comma in the number of reviews. First, we need to get rid of the comma, which we will do using str_replace(). We replace the comma with nothing as shown in the code below and then convert it into a number.

Real Price

The price is one of the most important factor when it comes to choosing a smart phone. Let us look at the price of the best selling mobile phones. Again, looking at the HTML code, the price can be located within the following:

• <span> tag identified by the class .a-text-strike
• section identified by the class crwPrice and .crwProductDetail

Using the above information, we can extract the price of the mobile phones which is returned as a character vector but we need to convert it to numeric if we are to analyze it further. Let use convert the price to a number using the following steps:

• use str_trim() to remove the white spaces
• exclude the currency information using str_sub()
• replace the comma using str_replace()
• remove the decimal values using str_split()
• extract the price from the resulting list using map_chr()
• convert the price to a number using as.numeric()

Actual Price

Deep discounts are one of the strategies adopted by ecommerce firms to drive sales. Let us look at the actual price (after discount price) of the best selling mobile phones. The discounted price can be located within the following:

• <span> tag identified by the class .crwActualPrice
• section identified by the class crwPrice and .crwProductDetail

Using the above information, we can extract the discounted price of the mobile phones. Let use convert the price to a number using the same steps as in the case of real price.

Title

As we did in the previous case study, we will look at the HTML code of the IMDB web page and locate the title of the movies in the following way:

• hyperlink inside <h3> tag
• section identified with the class .lister-item-content

In other words, the title of the movie is inside a hyperlink (<a>) which is inside a level 3 heading (<h3>) within a section identified by the class .lister-item-content.

Year of Release

The year in which a movie was released can be located in the following way:

• <span> tag identified by the class .lister-item-year
• nested inside a level 3 heading (<h3>)
• part of section identified by the class .lister-item-content

If you look at the output, the year is enclosed in round brackets and is a character vector. We need to do 2 things now:

• remove the round bracket
• convert year to class Date instead of character

We will use str_sub() to extract the year and convert it to Date using as.Date() with the format %Y. Finally, we use year() from lubridate package to extract the year from the previous step.

Certificate

The certificate given to the movie can be located in the following way:

• <span> tag identified by the class .certificate
• nested inside a paragraph (<p>)
• part of section identified by the class .lister-item-content

Runtime

The runtime of the movie can be located in the following way:

• <span> tag identified by the class .runtime
• nested inside a paragraph (<p>)
• part of section identified by the class .lister-item-content

Genre

The genre of the movie can be located in the following way:

• <span> tag identified by the class .genre
• nested inside a paragraph (<p>)
• part of section identified by the class .lister-item-content

Rating

The rating of the movie can be located in the following way:

• part of the section identified by the class .ratings-imdb-rating
• nested within the section identified by the class .ratings-bar

Since rating is returned as a character vector, we will use as.numeric() to convert it into a number.

XPATH

Votes

To extract votes from the web page, we will use a different technique. In this case, we will use xpath and attributes to locate the total number of votes received by the top 50 movies.

xpath is specified using the following:

• tab
• attribute name
• attribute value

In case of votes, they are the following:

• meta
• itemprop
• ratingCount

Next, we are not looking to extract text value as we did in the previous examples using html_text(). Here, we need to extract the value assingned to the content attribute within the <meta> tag using html_attribute().

Finally, we convert the votes to a number using as.numeric().

Revenue

We wanted to extract both revenue and votes without using xpath but the way in which they are structured in the HTML code forced us to use xpath to extract votes. If you look at the HTML code, both votes and revenue are located inside the same tag with the same attribute name and value i.e. there is no distinct way to identify either of them.

In case of revenue, the xpath details are as follows:

• <span>
• name
• nv

Next, we will use html_text() to extract the revenue.

To extract the revenue as a number, we need to do some string hacking as follows:

• extract values that begin with $
• omit missing values
• convert values to character using as.character()
• append NA where revenue is missing
• remove $ and M
• convert to number using as.numeric()